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1.  General  Summary  of  Achievements. 


The  grant  FA8655-09-1-3083,  '‘The  Relationship  Between  Visual  Sensor  Equipment 
in  Flying  Insects  and  their  Flight  Performance  -  a  ‘Neurobio-Engineering’ 
Approach",  that  was  awarded  to  support  my  research  and  the  work  of  Dr.  Sean 
Humbert,  (UMD,  FA95 5 0-09- 1-0075)  has  come  to  an  end.  In  this  final  report  on  the 
scientific  activities  in  my  lab  I  will  stay  with  the  tradition  to  provide  an  overview  of 
the  work  that  is  relevant  to  AFRL/AFOSR  -  which  basically  includes  most,  if  not  all 
projects  undertaken  in  my  lab  during  the  funding  period. 

Despite  the  fact  that  the  seasonal  availability  of  horseflies,  robberflies  and  hoverflies 
did  limit  progress  regarding  a  comprehensive  comparative  study  of  across  several 
dipteran  fly  species,  the  following  sections  should  demonstrate  that  we  made  some 
important  advances  regarding  the  biological  design  principles  of  multisensory  reflex 
control.  One  of  the  major  strategic  moves  was  to  focus  on  gaze  stabilization  as  an 
approximation  to  study  flight  control.  The  rational  being  that  head  movements 
induced  by  multiple  sensor  systems  compensate  for  unpredictable  disturbances  of  the 
thorax  flies  encounter  on  the  wing.  In  addition,  Sean  Humbert  showed  in  his  work  that 
a  head  orientation  which  enables  the  alignment  of  head-centred  sensor  systems  with 
the  inertial  vector  significantly  simplifies  the  measurement  of  state-changes  required 
for  feedforward/feedback-based  flight  control.  As  a  corollary  of  our  joint  studies  we 
should  state:  Flying  insects  which  are  subject  to  significant  inertial  forces  during 
flight  and  have  limited  computing  power,  gaze  stabilization  is  a  functional  necessity 
that  enables  high  aerial  manoeuvrability. 

Because  Sean  Humbert  has  submitted  a  separate  final  report  on  his  efforts,  I  will 
focus  here  on  activities  in  my  lab,  adding  occasional  cross-references  to  his  report 
where  appropriate. 


1.1  Publications 

During  the  funding  period  of  this  grant  from  Oct  2009  until  the  end  of  Sept  2013  my 
group  has  published  13  papers  in  peer  reviewed  journals  (032-45),  3  book  chapter 
(R4-6),  and  5  peer-reviewed  conference  papers  (C4-9).  Six  of  the  peer-reviewed 
papers  were  published  in  high  profile  journals  ranked  ‘internationally  competitive’. 
Finally,  I  am  a  co-editor  of  last  year’s  proceedings  of  the  ‘Living  Machine 
Conference’  on  biomimetic  and  biohybrid  systems. 

We  are  currently  working  four  further  scientific  publications  one  of  which  will  be 
submitted  within  the  next  week  (046),  with  the  others  to  follow  within  the  next  month 
(047-49). 


1.2  Scientific  Workshops,  Conferences,  and  Seminars 

In  total  I  attended  22  scientific  events  within  the  duration  of  this  grant  across  a  wide 
range  of  subject  areas  from  the  five-yearly  specialized  International  Conferences  on 
Invertebrate  Vision  to  the  novel  annual  Living  Machine  Conference  which  is 
dedicated  to  bio-inspired  technology.  At  three  of  those  conferences  Dr  Kit  Longden, 
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the  RA  on  this  grant,  also  presented  his  work.  In  addition  I  was  invited  to  give 
seminar  presentations  in  continental  Europe  and  in  the  UK  at  7  occasions. 

Many  of  the  conferences  were  in  connection  with  the  bio-MAV/UAS  SOAR 
meetings,  bio-inspired  GNC  at  Eglin  Air  Force  Base,  or  other  events  relevant  to  the 
Programme  Arrangement  between  AFRL/AFOSR  and  DSTL  on  bio-inspired 
technology. 


1.3  Poster  and  Oral  Presentations  at  National  and  International  Conferences 

Members  of  my  group  presented  14  poster  (A43-54)  and  oral  contribution  at  7 
national  and  international  conferences. 

1.4  Funding 

Current  funding  secured  for  research  in  my  laboratory  (excluding  FA8655-09-1- 
3083): 

•  Wellcome  Trust  research  grant,  PI:  “Integrated  reflex  control”  in  Manduca. 

[Funding  until  end  of  April  2014] 

•  DSTL-sponsored  EPSRC  Industrial  CASE  PhD  Studentship,  PI:  “The 
significance  of  (image)  gaze  stabilization  -  Comparative  studies  on  gaze 
control  design  in  flying  insects  using  behavioural,  computational,  and 
electrophysiological  techniques.  ”  [Funding  until  end  of  March  2015] 

•  DSTL  National  PhD  Programme,  PI:  “The  significance  of  (image)  gaze 
stabilization  -  functional  characterization  of  the  neck  motor  system  using 
electrical  stimulation  of  identified  neck  muscles  in  blowflies.  ”  [Funding  until 
end  of  September  2016] 


Grant  proposals  submitted: 

•  AFOSR/EOARD,  PI,  together  with  Dr  Graham  Taylor,  PI,  University  of 
Oxford,  UK,  and  Dr  Sean  Humbert,  PI,  University  of  Maryland:  “The  mode 
sensing  hypothesis  ”  [F‘  year  funding  currently  being  set  up  at  UMD  ] 

•  French-UK  PhD  studentship  scheme  (DSTL),  PI,  together  with  Dr  Stephane 
Viollet,  CNRS,  Marseille,  FR:  “Head-body  movements  in  freely  flying  flies 
when  chasing  a  dummy  target” 


Grant  proposals  in  preparation: 

•  AFOSR/EOARD,  PI,  together  with  Prof  Doekele  Stavenga,  University  of 
Groningen,  NL:  “Polarization  Vision  in  Horseflies” 
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1.5  Group  size,  Collaborations,  and  Scientific  Impact 

Group  size: 

2  RAs  (postdocs),  3  postgraduate  students  (PhD),  1  MSc  and  5  undergraduate 
students.  I  co-supervise  a  DSTL-funded  DPhil  student  working  in  Graham 
Taylor’s  lab.  University  of  Oxford,  on  the  mode  sensing  hypothesis  in  Manduca 
sexta,  and  a  PhD  student  in  Barbara  Webb’s  lab,  University  of  Edinburgh,  funded 
by  the  BBSRC  who  works  on  visual  navigation  in  ants. 


Collaborations: 

I  have  collaborations  at  Imperial  College  in  the  Department  of  Bioengineering 
with:  Dr  Martyn  Boutelle  (Biosensors),  Dr  Simon  Schultz  (Neural  Coding),  Dr 
Manos  Drakakis  (Low-power  VLSI  technology),  and  Dr  Reiko  Tanaka 
(Compound  Control).  To  study  the  functional  anatomy  of  insects  we  still 
collaborate  with  the  micro-CT  group  at  the  Natural  History  Museum,  London. 
Our  collaboration  with  scientists  at  the  Paul  Scherre  Institute,  Swiss  Light 
Source,  CH,  to  obtain  high-resolution  4D  data  from  tethered  flying  flies  on  the 
functional  organization  of  their  fight  and  neck  motor  systems  is  ongoing  -  we  are 
currently  publishing  the  results  of  to  beam  time  grants  I  got  awarded  in  the  past. 
Collaborations  with  Prof  Simon  Laughlin  (Cambridge)  and  Dr  Graham  Taylor 
(Oxford)  have  been  maintained.  Work  with  Graham  Taylor  and  Prof  Sean 
Humbert  (UMD,  USA)  will  be  intensified  by  means  of  the  AFOSR-funded  grant 
on  the  ‘mode  sensing  hypothesis’.  A  new  collaboration  has  now  been  established 
with  Prof  Barbara  Webb  to  study  ant  navigation.  Further  international 
Collaborations  include  work  with  Prof  Martin  Egelhaaf,  Neurobiology,  Bielefeld 
University,  Germany;  Prof  Fabrizio  Gabbiani,  Baylor  College  of  Medicine,  USA; 
and  Mr  Ric  Wehling  at  the  AF  Research  Laboratory,  Eglin,  US.  A  new 
collaboration  with  Prof  Doekele  Stavenga,  University  of  Groningen,  NL,  and  Dr 
Stephane  Viollet,  CNRS,  Marseille,  FR  are  in  planning. 


Current  Bibliometric  Data  and  Scientific  Impact: 

Bibliometrics: 


March  2014 

Number  of  citations* 

2137 

h-factor* 

25 

*  =  according  to  “Google  Scholar 


Scientific  Impact/Leadershw: 

•  Reviewing  manuscripts  for  more  than  30  peer  review  journals,  including 
Nature,  Science,  and  Neuron. 


4 


Distribution  A:  Approved  for  public  release;  distribution  is  unlimited. 


•  Reviewing  grant  applications  for  10  funding  agencies  including  RCUK,  NSF 
(USA),  andAFOSR  (USA). 

•  Expert  Reviewer  for  European  Commission  FP7  collaborative  project 

•  Member  of  Academic  Editorial  Board  of  open  access  journal  PLoS  ONE 

•  Financial  Co-organizer  AFOSR/DSTL  “5/o  MA  V  SOAR”  meeting,  Chilworth 
Manor,  2011,  as  well  as  UK  follow-up  meetings  at  Imperial  College,  London, 
and  University  of  Oxford,  2012. 

•  Member  of  Programme  Committee  of  Conference  on  Biomimetic  and 
Biohybrid  Systems  “Living  Machines”,  Barcelona  and  London  2012  and  2013 

•  Co-ordination  of  DSTL-supported  virtual  Centre  of  Excellence  on  Unmanned 
Autonomous  Systems  (UAS -former  “MAY”) 

•  Local  Co-organizer  (Satellite  events  chair)  of  Conference  on  Biomimetic  and 
Biohybrid  Systems  “Living  Machines”,  London,  2013 

•  Co-applicant/proposer  of  research  call  initiatives  in  the  area  of  sensing  and 
actuation  (ESF/NSF)  and  ‘Robot  Companions”,  Flagship  proposal  submitted 
to  the  European  Commission. 

1  also  took  on  leadership  responsibility  in  fostering  areas  relevant  to  the  remits  of 

AFRL/AFOSR,  contributing  to  high  standards  in  scientific  research  by 

committing  to  reviewing  and  editorial  work,  as  well  as  applying  for  an  EPSRC 

Centre  of  Doctoral  Training  in  Bio-inspired  Systems  and  Technologies,  strongly 

supported  by  DSTL. 


2.  Report  on  current  projects: 

I  will  outline  the  achievements  during  the  funding  period  of  this  grant  (FA8655-09-1- 
3083)  in  relation  to  the  four  major  directions  of  my  work: 

2.1.  Behavioural  performance  of  multisensory  motor  control  in  dipteran  flies 

2.2.  Biomechanics  of  motor  systems  involved  in  gaze  and  flight  control 

2.3.  Neural  mechanisms  underlying  multisensory  motor  control  in  dipteran  flies 

2.4.  Modelling  of  multisensory  motor  control  design 

These  four  approaches  result  in  a  comprehensive  description  of  sensorimotor  control. 
The  results  obtained  at  different  system  levels  and  by  applying  a  variety  of 
experimental  and  theoretical  approaches  do  inform  and  support  each  other.  Altogether 
they  enable  an  iterative  cross-validation  process  that  facilitates  the  derivation  of 
models  capturing  the  fundamental  design  principles  of  biological  sensorimotor 
control. 
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Although  funding  for  my  research  over  the  past  nine  years  was  also  provided  by  the 
the  BBSRC,  Human  Frontier  Science  Program  Trust,  Wellcome  Trust,  DSTL  and 
Imperial  College  resources,  I  will  briefly  summarize  earlier  findings  (presented  in 
earlier  reports)  and  the  development  over  the  last  couple  of  months  followed  by  a 
conclusion. 


2.1.  Behavioural  performance 

2-degrees  of  freedom  controller 

We  applied  a  linear  systems  analysis  approach  to  the  gaze  stabilization  system  in 
dipteran  flies  to  quantify  the  interaction  between  the  compound  eyes  (mediating  optic 
flow  information),  the  ocelli  (mediating  information  on  fast  changes  in  attitude)  and 
the  mechanosensory  halteres  (mediating  rotation  rates)  in  the  high  dynamic  input 
range. 

In  our  original  studies  on  Calliphora  we  performed  a  systems  identification  of  the 
gaze  stabilization  system  which  showed  a  surprisingly  linear  performance  under  the 
conditions  tested  (C4).  An  intermodulation  distortion  (IMD)  analysis  indicated  that 
internal  non-linearities  in  the  system  contribute  less  than  4%  to  the  output  signal. 

Figure  1:  2-degree  of  freedom  controller 
combining  feedforward  signals  provided  by 
the  halteres  (Fff)  with  feedback  signals 
provided  by  the  motion  vision  pathway  (Ffb). 
The  properties  of  the  neck  motor  system  are 
included  by  the  transfer  function  of  the  neck 
motor  system  (Fn).  For  further  details  see 
text.  Figure  modified  from  (C4). 


Transfer  function 

HR  _  _  F„  ftfii 

TR  -  "  ^*F^F„ 


The  resulting  control  architecture  and  transfer  functions  describing  the  relationship 
between  the  stimulus  (thorax  rotation,  TR)  and  the  systems  output  (head  rotation,  HR) 
are  shown  in  Fig  1 . 

Recent  work  on  the  interpretation  of  the  architecture  and  further  analysis  of  the  data 
revealed  that  the  system  including  the  halteres  and  the  compound  eyes  can  be 
described  as  a  2-degrees  of  freedom  controller  where  1  degree  of  freedom  is  given  by 
a  feed  forward  component  {Fyy  halteres)  and  the  second  degree  of  freedom  represents 
a  feedback  component  (Fp,,  compound  eyes).  This  ingenious  combination  of  fast  feed 
forward  signals  mediated  by  the  halteres  and  slow  feedback  signals  from  the 
compound  eyes  works  as  follows:  immediate  haltere-induced  compensatory  head  roll 
shifts  the  retinal  slip  speed  distribution  caused  by  thorax  rotation  towards  to  lower 
dynamic  range  (subtraction  of  initial  head  roll  from  the  thorax  roll)  that  can  then  be 
analysed  by  the  motion  vision  pathway  of  the  compound  eyes.  Thus,  the  halteres 
‘enable’  the  visual  system  by  eliminating  those  fast  slip  speed  components  the  visual 
system  would  not  be  able  to  process  due  to  its  band-pass  characteristics. 
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Figure  2:  Probability  density  functions 
of  retinal  slip  speeds  under  three 
experimental  conditions  and  at  three 
different  thorax  oscillations,  3  Hz 
(upper  panel),  6  Hz  (middle  panel),  and 
15  Hz  (bottom  panel).  The  three  traces 
in  each  panel  show  the  distribution 
under  condition  where  the  head  would 
be  fixed  to  the  thorax  (HR  =  TR,  blue), 
the  halteres  and  the  motion  vision 
pathway  (green),  and  only  the  motion 
vision  pathway  (red)  contribute  to 
compensatory  head  roll.  Note  that  the 
contribution  of  the  halteres  to  shift  the 
slip  speed  probability  density  function 
depends  on  the  stimulus  frequency.  At 
the  highest  stimulation  frequency  halter 
function  is  essential  for  enabling  the 
processing  of  visual  motion.  If  the 
halteres  are  disabled  visual  feedback 
causes  a  shift  of  the  density  function  to 
higher  slip  speeds,  which  has  a 
detrimental  effect  on  gaze  stabilization, 
(unpublished  data). 


The  data  presented  in  Figure  2  demonstrate  the  prineiple.  They  show  probability 
density  functions  of  the  retinal  image  shifts  (slip  speed)  the  fly  encounters  during 
different  experimental  conditions,  i.e.  the  frequency  of  occurrence  plotted  against  slip 
speed.  Three  different  oscillation  frequencies  of  the  thorax  have  been  applied,  3  Hz,  6 
Hz,  and  15  Hz.  The  blue  traces  in  the  subpanels  indicate  the  distribution  the  fly  would 
experience  if  its  head  was  was  fixed  to  the  thorax.  The  green  traces  correspond  to  the 
distribution  when  the  head  is  free  to  move  and  both  the  compound  eyes  and  the 
halteres  are  contributing  to  the  compensatory  head  movements  (+  Hs).  In  this  case  the 
slip  speed  distribution  is  shifted  to  the  left,  i.e.  to  lower  slip  speeds.  Red  traces  show 
the  shift  of  the  distribution  after  the  halteres  were  removed.  At  thorax  oscillations  of 
15  Hz  the  significance  of  the  feed  forward  component  becomes  most  obvious  (Fig  2, 
bottom  panel).  With  haltere  and  compound  eye  contributions  the  slip  speed 
distribution  is  markedly  shifted  to  the  left  (green  trace)  while  without  halters  (red 
trace)  a  shift  of  the  distribution  to  the  right  is  observed.  These  data  demonstrate  a  co¬ 
operative  effect  of  the  two  control  degrees  of  freedom  that  is  different  from  just 
adding  up  the  signals  generated  by  the  compound  eyes  and  the  halteres  alone. 

This  interpretation  is  in  agreement  with  the  qualitative  finding  that  flies  the  halteres  of 
which  have  been  removed  will  immediately  crash  if  thrown  in  the  air.  Only  if  a  light 
weight  thread  is  attached  to  their  abdomen,  they  are  able  to  fly  again.  The  thread,  in 
this  case  adds  passive  aerodynamic  stability  and  dampens  fast  attitude  changes 
resulting  in  high  retinal  slip  speeds.  With  fast  retinal  slip  speeds  removed  the  visual 
system  is  sufficient  to  provide  feedback  signals  for  flight  control. 
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A  necessary  condition  for  this  2-D  controller  to  work  is  that  the  motion  vision 
component  of  the  controller  provides  stable  feedback.  We  assessed  the  stability 
margins  of  the  feedback  controller  and  found  that  both  the  gain  and  the  phase  margins 
are  unexpectedly  high  with  20  dB  and  130  degrees,  respectively.  Our  interpretation  of 
those  findings  is  that  the  system  is  specified  to  deal  with  a  high  variance  of 
unpredictable  input  signals  which,  in  combination  with  noise  in  the  sensory  pathways, 
potentially  results  in  extreme  output  values  without  becoming  instable. 

We  are  currently  writing  up  our  work  on  the  2-degree  of  freedom  controller  for 
publication  (049). 


Studies  on  gaze  stabilization  in  robberflies 

The  supply  of  robberflies  from  the  US  turned  out  to  be  difficult.  We  observed  a  severe 
reduction  of  the  life  expectancy  during  intercontinental  transportation  by  air,  probably 
because  the  fly  respiratory  system  does  not  tolerate  the  massive  pressure  changes  in 
the  cabin  of  air  carriers.  We  therefore  collected  two  UK  robberfly  species  in  the  south 
of  England,  Dysmachus  trigonus  and  Philonicus  albiceps.  The  former  species  was 
used  in  preliminary  electrophysiological  experiments  (see  below),  while  the  latter  had 
been  used  for  behavioural  experiments  to  study  gaze  stabilization,  applying  the  same 
linear  systems  approach  as  to  blowflies  the  results  of  which  were  reported  earlier. 

Robberfies  are  more  difficult  to  study  under  tethered  flight  conditions.  They  do  not 
assume  the  same  stable  flight  position  a  Calliphora  and  often  extend  their  legs  to 
counteract  external  body  rotations.  We  were  nonetheless  able  to  perform  some 
preliminary  gaze  stabilization  experiments  on  P.  albiceps  which  provided  us  with 
some  insight  regarding  general  principles  and  species-specific  adaptations. 

Figure  3  shows  the  data  obtained  from  at  least  nine  robberflies  the  thorax  of  which 
was  oscillated  over  the  range  from  1-lOHz  at  an  amplitude  +/-  30°.  The  bode  plots 
shown  were  obtained  under  four  different  conditions  -  the  same  that  had  been  applies 
to  blowflies  in  previous  studies  -  in  terms  of  sensory  inputs:  Cl  (orange  trace)  all 
sensory  inputs  intact  (n=l  1),  C2  (dark  blue)  compound  eyes  and  ocelli  (n=9),  C3  (red) 
compound  eyes  and  halteres  (n=l  1),  and  C4  (light  blue)  only  compound  eyes  (n=9). 

The  most  conspicuous  difference  in  the  gaze  stabilization  performance  compared  to 
the  control  condition  where  all  sensors  are  intact  was  observed  after  the  removal  of 
the  halteres;  a  similar  result  was  found  in  blowflies  (C4).  Removing  the  halteres 
drastically  increase  the  phase  delay  of  the  response  (blue  traces,  bottom  plot)  -  in 
particular  when  the  ocellar  input  is  disabled  (light  blue  trace).  The  response  gain 
shows  no  statistically  significant  differences,  except  for  the  condition  where  both  the 
haltere  and  ocellar  inputs  are  excluded.  Any  differences  between  the  responses  under 
the  four  conditions  measured  in  the  low  dynamic  range  (1-3  Hz)  are  probably  due  to 
high  interspecific  variability.  Overall,  the  results  obtained  in  P.  albicepts  are 
comparable  with  those  obtained  in  Calliphora.  In  both  species  haltere  input  reduces 
the  phase  delay  of  the  response  (orange  and  red  traces,  bottom  plot). 
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Figure  3:  Frequency  response  of  compensatory  head  roll  in  P.  albiceps.  Top  plot:  Head  roll 
gain  against  oscillation  frequency  for  four  different  combinations  of  sensory  inputs  (see  inset, 
bottom  plot).  Probably  due  to  a  large  response  variability  across  animals  no  significant 
differences  were  found  for  three  out  of four  conditions.  When  only  motion  vision  contributes 
to  the  response  the  gain  is  significantly  reduced  for  an  oscillation  frequency  of  10  Hz.  Bottom 
plot:  The  head  roll  phase  angle  relative  to  the  phase  of  the  stimulus  as  a  function  of  stimulus 
frequency.  Haltere  contribution  results  in  a  minimal  phase  delay  over  the  entire  dynamic 
range  tested.  There  is  a  significant  difference  in  phase  delay  between  responses  obtained 
without  halteres  (blue  traces)  and  all  other  stimulus  conditions.  The  largest  phase  delay  is 
observed  when  only  the  motion  vision  pathway  contributes  to  compensatory  head  roll  (light 
blue  trace).  (Unpublished  data). 


In  a  direct  comparison  between  the  gain  of  compensatory  head  movements  measured 
in  blowflies  and  robberflies  mediated  only  by  the  motion  vision  pathway  (ocelli 
covered  and  halteres  removed)  we  found  that  compensatory  head  roll  in  Philonicus 
covers  a  higher  dynamic  input  range  compared  to  Calliphora  (Figure  4).  These  results 
could  suggest  that  the  motion  vision  pathway  in  robberflies  is  tuned  to  higher  angular 
velocities.  But  it  could  also  mean  that  the  neck  motor  system  of  the  two  species  has 
different  dynamic  properties.  The  latter  aspect  would  be  supported  by  the  fundamental 
difference  the  anatomical  constraints  of  the  neck  motor.  In  robberflies  the  head  is 
positioned  more  distantly  from  the  thorax  supporting  larger  head  movements.  Further 
functional  anatomical  (see  below)  and  behavioural  studies  will  be  required  before  our 
results  can  be  properly  interpreted. 
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Figure  4:  Gain  of  compensatory  head  roll  in  robberflies  (n=9)  and  blowflies 
(n=7)  mediated  by  the  motion  vision  pathway.  The  thorax  of  the  flies  was 
oscillations  at  +/-  30°  at  different  frequencies.  At  high  stimulus  frequencies 
robberflies  show  a  significantly  higher  gain.  Error  bars  give  SEM  (unpublished 
data). 


Interactions  between  motion  vision  pathway  and  ocelli 

Since  the  work  by  Parsons  et  al.  (020,  034)  we  know  that  the  LPTCs  (VS-cells)  are 
modulated  by  ocellar  stimulation  suggesting  both  pathways  to  use  a  common  sensory 
coordinate  system,  likely  to  be  set  up  by  the  LPTCs  (026). 

One  of  the  major  objectives  of  this  project  was  to  study  the  interaction  between  the 
compound  eye-mediated  motion  vision  pathway  and  the  ocellar  pathway  in  the  gaze 
stabilization  system  of  different  dipteran  species.  Although  the  availability  of  species 
other  than  our  lab-bred  blowflies  was  rather  limited  throughout  the  duration  of  the 
grant,  we  did  achieve  comparative  measurements  on  blowflies  and  robberflies. 

In  a  first  set  of  experiments  we  oscillated  a  dark  visual  hemisphere  as  a  substitute  for 
ground  around  tethered  flying  blowflies.  Under  these  conditions  the  animals  had 
access  only  to  visual  input,  but  no  haltere  information,  where  the  ocelli  were  either 
covered  (compound  eyes)  or  uncovered  (compound  eyes  +  ocelli).  The  results  in 
terms  of  compensatory  head  roll  are  shown  in  Figure  5.  We  found  that  under  these 
two  conditions,  over  the  range  of  oscillation  frequency  from  1  -  10  Hz  there  was  no 
significant  difference  between  the  response  gain  observed.  The  phase,  however, 
showed  a  clear  advance  of  the  response  when  the  ocelli  were  contributing  for 
frequencies  >  3  Hz.  This  result  suggests  a  non-linear  integration  of  ocellar  and  motion 
vision  signals. 
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Figure  5:  Compensatory  head  roll  mediated  by  compound  eye  and  compound  eye  +  ocelli 
information.  Left  panels:  Linear  response  gain  (upper  panel)  and  response  phase  (lower 
panel)  plotted  as  a  function  of  pattern  oscillation  frequency.  Note  that  there  is  no  significant 
difference  between  the  condition  with  and  without  ocelli  regarding  the  response  gain  while 
the  response  phase  becomes  larger  for  oscillation  frequencies  >  3  Hz.  Right  panels:  The 
simulation  of  the  ocelli  and  compound  eye  as  simple  highpass  (tau  =  12.5ms)  and  lowpass 
filters  (tau  =  12.5ms),  respectively,  combined  with  an  'integrate  and  fire  model’  of  the 
descending  neuron  that  included  a  static  threshold  non-linearity  qualitatively  captures  the 
behavioural  results,  (unpublished  results). 


We  combined  a  phenomenological  approximation  of  the  two  sensor  system  properties 
by  means  of  a  low  pass  (motion  vision)  and  a  high  pass  (ocelli)  filter,  with  time 
constants  in  the  physiologically  plausible  range,  and  a  biophysically  motivated  leaky 
integrate  and  fire  model.  A  biophysically  motivated  ‘integrate  and  fire  model’  was 
implemented  to  compute  the  membrane  potential  of  a  descending  neuron  that 
integrates  the  outputs  from  both  sensor  systems  (Figure  6).  Like  all  ‘real’  nerves  cells 
the  descending  neuron  incorporates  a  static  threshold  non-linearity.  Only  beyond  a 
certain  threshold  membrane  potential  does  the  descending  neuron  generate  action 
potentials,  which  are  necessary  to  produce  muscle  contractions  in  the  neck  motor. 


input 


compound 

eyes 


ocelli 


Figure  6:  A  model  that  qualitatively  explains  the  observed  phase  increase  in  compensatory 
head  roll  of  Calliphora  when  the  ocelli  are  disabled  while  the  gain  stays  the  same  for 
compound  eye  and  compound  eye  +  ocelli  stimulation,  (unpublished). 
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This  model  qualitatively  reproduces  the  effect  observed  at  the  behavioural  level: 
while  the  gain  is  the  same  for  both  experimental  conditions,  the  phase  of  the 
responses  at  higher  stimulus  frequencies  is  advanced  if  the  ocelli  contribute  (lower 
right  panel).  This  comparatively  simple  model  suggests  a  static  threshold  non-linearly 
as  the  potential  basis  for  the  observed  behavioural  data. 


Gain  and  phase  of  closed-loop  head  roll  response,  N  =  9.  stimulus  amplitude  30" 


Frequency  [Hz] 


Figure  7:  First  studies  on  the  interaction  of  ocelli  and  compound  eye  information  in  the 
robberfly  Dysmachus  trigonus  (left  panel).  In  these  experiments  the  halteres  of  the  fly  were 
removed  and  rather  than  oscillating  the  ground  pattern  the  thorax  of  the  animals  were 
sinusoidally  rotated  around  the  roll  axis.  The  main  senor  systems  contributing  to  the  response 
were  therefore  the  compound  eyes  or  the  compound  eyes  +  ocelli.  Right  panels:  Similar  to  the 
experiments  in  Calliphora,  the  gain  was  similar  under  the  two  conditions  (upper  plot)  but 
there  was  a  difference  in  the  response  phase  (lower  plot),  (unpublished  data). 

We  have  also  started  performing  specific  experiments  on  ocellar  and  compound  eye 
signal  integration  to  study  gaze  stabilization  in  UK  robberfly  species  Philonicus 
albiceps  and  Dysmachus  trigonus.  The  behavioural  results  indicate  that  similar  non- 
linearities  as  described  for  blowflies  above  may  apply  to  the  gaze  stabilization 
systems  in  robberflies.  Whether  or  not  a  static  threshold  non-linearity  at  the  neuronal 
level  best  explains  our  findings  regarding  ocellar  and  compound  eye  signal  integration 
across  all  dipteran  species  is  not  entirely  clear  at  this  point. 

Next  month  we  will  perform  more  experiments  on  Calliphora  -  and  as  soon  as  they 
will  emerge  in  the  wild,  on  robberflies,  horseflies  and  hoverflies  -  to  substantiate  our 
data  base.  We  will  also  refine  our  open  loop  integration  model.  The  model  will  be 
included  as  a  first  non-linear  element  in  our  otherwise  linear  closed-loop  simulation 
platform  of  the  gaze  stabilization  system  (cf  section  2.4)  before  writing  up  the  results 
for  publication. 

Altogether,  the  results  of  our  behavioural  experiments  on  gaze  stabilization  in 
dipteran  flies  proved  to  be  a  worthwhile  paradigm  to  study  the  relationship  between 
sensory  and  motor  coordinate  systems.  We  expect  further  advances  regarding  a 
quantitative  description  of  the  weighting  given  to  the  various  sensor  systems  by 
applying  band-limited  white  noise  stimuli  with  different  statistical  properties  and  by 
new  experiments  where  pattern  and  body  rotations  may  be  combined  in  different 
ways. 
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2.2.  Biomechanics  of  motor  systems 

The  neck  motor  system 


A  comprehensive  understanding  of  the  relationship  between  sensor  and  motor 
coordinate  systems  requires  a  functional  characterization  of  the  motor  system.  To  this 
end  we  are  combining  functional  anatomy  based  on  x-ray  techniques  such  as  p-CT 
with  electrical  muscle  stimulations  and  biomechanical  modelling. 

Based  on  previous  research  in  collaboration  with  the  Natural  History  Museum 
London,  we  are  currently  advancing  methods  to  increase  tissue  contrast  in  p-CT 
scans  to  enable  automatic  segmentation  of  different  tissue  types.  Our  first  results,  still 
based  on  manual  segmentation,  provided  us  with  a  3 -dimensional  reconstruction  of 
the  neck  motor  system  in  the  blowfly  Calliphora  which  will  serve  two  purposes:  (i) 
the  3D  data  allow  us  to  refine  our  dissection  methods  in  terms  of  minimizing 
structural  damage  to  the  system  when  placing  electrodes  for  electrical  muscle 
stimulation  (ii)  it  will  inform  biomechanical  models  of  the  neck  motor  system,  e.g. 
Finite  Element  Models,  which  capture  the  dynamic  properties  of  the  system. 
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Figure  8:  A  quantitative  and  qualitative  comparison  of  the  5  preparation  methods  for  the  final 
staining  day.  A:  A  comparison  of  day  7  contrast  ratios  across  the  5  staining  protocols,  for  tissue  types 
(inset).  B-E:  Coronal  slices  of  ETH-PTA  (B),  PFA-PTA  (C),  ETH-Lugol  (D),  PFA-Iodine  (E)  at 
approximately  the  same  depth  and  orientation.  The  slice  for  ETH-Iodine  exhibits  similarly  poor 
contrast  to  that  of  PFA-Iodine  and  is  omitted,  (data  from  048) 
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Figure  8  shows  some  of  the  results  we  obtained  in  a  study  using  different  fixatives 
(e.g.  ETH  =100  ethanol,  PFA  =  paraform  aldehyde)  chemical  agents  (e.g.  iodine, 
PTA  =  phosphotungstic  acid)  and  staining  protocols  to  increase  the  tissue  contrast.  In 
these  experiments  we  did  not  optimize  for  the  highest  voxel  resolution,  which  was  just 
below  10  pm^,  but  to  increase  the  tissue  contrast  ratio.  Our  data  show  that  the  best 
ratios  are  obtained  using  a  combination  from  ETH  or  PFA  as  a  fixative  and  PTA  as  a 
contrast  agent.  We  are  currently  preparing  a  manuscript  for  publication  reporting  the 
results  on  our  systematic  study  to  improve  the  p-CT  methodology  (048)  for  the 
functional  analysis  of  insect  motor  systems. 

In  earlier  work  reported  previously  we  applied  iodine  as  a  contrast  agent  to 
reconstruct  the  3 -dimensional  organization  of  the  neck  motor  system.  As  mentioned 
above,  the  results  were  meant  to  develop  new  dissection  methods  applied  in  the 
context  of  electrical  stimulations  of  neck  muscles  to  study  their  dynamic  properties 
(Figure  9). 


Figure  9:  g-CT-based  reconstruction  of  the  neck  motor  system  in  Calliphora.  From  left  to 
right:  right,  top  and  left  view  of  the  blowfly  head.  False  colours  mark  muscles  attached  to 
cuticular  structures  in  the  thorax  and  the  head  enabling  gaze  stabilization,  (unpublished 
data). 

The  latest  advances  in  tissue  contrast  presented  above  will  allow  us  to  refine  the 
current  3-dimensional  anatomical  model  of  the  Calliphora  neck  motor  system.  We  are 
also  planning  to  obtain  similar  data  from  other  dipteran  species  for  a  comparative 
study. 

A  key  component  of  our  work  will  be  the  specification  of  a  biomechanical  model  that 
captures  the  properties  of  the  neck  motor  system.  We  decided  to  apply  finite  element 
modelling  (FEM)  to  the  system  as  it  provides  a  convenient  tool  to  describe  and 
analyse  the  function  of  complex  motor  system.  The  FEM  will  be  informed  by  our  3- 
dimensional  anatomical  data  -  and  later  by  electrical  stimulation  experiments. 
Currently,  we  base  the  model  on  assumptions  regarding  the  local  Young’s  modulus. 
Despite  this  initial  simplification  of  the  system’s  functionality,  we  have  been 
successful  to  generate  head-body  movements  by  introducing  local  forces  simulating 
muscle  contraction  (Figure  10). 

The  combined  approaches  of  physiological  characterization  and  modelling  of  the 
system’s  properties  will  mutually  support  each  other.  Once  a  sufficiently  detailed 
model  is  derived  and  validated,  a  dimensionality  reduction  will  be  performed  to 
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obtain  a  simplified  version  of  a  neek  motor  system  that  may  be  implemented  in 
teehnieal  applieations. 


Figure  10:  Finite  model  of  the 
Calliphora  neck  motor  system 
based  on  3-dimensional 
anatomical  data  (ventral 
view).  Each  element 
describing  part  of  a  neck 
muscle  is  assigned  a  Young's 
modulus  of  a  certain 
orientation.  Simulating  forces 
in  the  different  muscles 
introduces  relative  movements 
between  head  and  body, 
(unpublished  data). 


The  flight  motor  system 

Together  with  Graham  Taylor’s  group  in  Oxford  and  a  group  of  seientists  at  the  Paul 
Seherrer  Institut  at  the  Swiss  Light  Souree  we  have  advaneed  x-ray  seanning 
teehniques  in  terms  of  spatial  and  temporal  resolution  in  a  way  that  enabled  us  to 
monitor  the  4-dimensional  operation  of  the  flight  motor  in  blowflies.  The  animals 
were  subjeeted  to  hard  x-ray  radiation  while  being  rotated  around  the  roll  axis  in 
tethered  flight.  Under  these  eonditions  the  flies  were  able  to  survive  for  several 
minutes  and  engaged  on  of  showed  asymmetrie  wing  trajeetories  in  an  attempt  to 
eompensate  for  the  externally  foreed  rotation  (Figure  11). 


Figure  11:  Schematic  illustration 
of  Calliphora  observed  in  a 
synchrotron  during  tethered  flight. 
The  fly  responds  to  a  forced  body 
rotation  by  generating  a 
differential  output  of  the  flight 
motor  -  here  by  increasing  and 
decreasing  lift  production  in  the 
left  and  right  wing,  respectively. 
The  X-  and  z-axes  correspond  to 
the  longitudinal  and  vertical  body 
axes  of  the  fly.  (figure  taken  from 
044). 


Due  to  retrospeetive  gating  of  images  phase  loeked  to  the  wing  beat  eyele  we  were 
able  to  observed  the  aetivity  of  the  power  and  steering  museles  during  the 
eompensatory  flight  manoeuvres.  Figure  12  show  an  external  view  of  the  thorax  and  a 
view  from  the  same  perspeetive  after  virtually  removing  large  parts  of  the  eutiele. 
Phase  eontrast  -  as  opposed  to  absorption  eontrast  in  p-CT  -  was  used  sueeessfully  to 
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resolve  even  the  buckling  of  tendons  of  steering  muscles  throughout  the  wing  beat 
cycle. 


Figure  12:  External  and  internal  view  of  the  Calliphora  thorax.  Left:  External  view  of  the 
thorax  showing  cuticular  structures  of  the  wing  hinge  and  parts  legs.  Right:  Virtual  dissection 
of  the  thorax  showing  the  power  muscles  of  the  flight  motor  (yellow,  orange,  and  red)  as  well 
as  four  small  identified  steering  muscles  the  activation  of  which  could  be  analysed  during 
asymmetric  tethered  flight  in  a  synchrotron.  (Figure  from  044). 


This  technique  allows  us  to  analyses  periodic  movement  patterns  in  any  motor  system 
of  flying  insects  as  long  as  an  appropriate  gating  signal  is  available  that  is  correlated 
with  the  movement.  Given  the  high  temporal  and  spatial  resolution  of  the  technique, 
in  further  studies  we  will  be  able  to  extract  general  principles  of  non-linear  and  linear 
operations  in  biological  motor  systems  which  will  inform  the  development  of  bio¬ 
mechanical  models.  In  combination  with  a  better  understanding  of  the  neural 
mechanisms  underlying  these  models  will  inform  novel  approaches  to  the  design  of 
small  scale  mechanical  devices  with  strong  potential  for  application  in  autonomous 
robotic  systems. 

On  our  highly  successful  synchrotron  work,  that  was  initiated  by  Daniel  Schwyn,  a 
former  PhD  student  in  the  lab,  we  published  two  up-front  reports  in  technical  journals 
(041  and  043)  last  year.  A  high  profile  publication  of  our  results  on  the  flight  motor 
system  is  now  accepted  for  publication  in  PLoS  Biology  (044)  with  another  one 
focusing  on  the  ground-breaking  new  methodology  currently  being  prepared  (046). 


2.3.  Neural  mechanisms  underlying  multisensory  motor  control  in 
dipteran  flies 

Optic  flow  processing  depends  on  locomotor  and  nutritional  state 

A  question  of  increasing  interest  concerns  a  notorious  problem  of  biological  systems 
that  is  even  as  significant  for  micro  air  vehicles  with  small  payloads:  limited  energy 
supply. 
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Neural  information  processing  is  costly  and  requires  sensory  systems  to  adjust  energy 
expenditure  very  tightly  to  the  current  needs.  Work  in  our  lab  has  shown  that  the 
activity  of  LPTCs  in  the  motion  vision  system  is  increased  during  locomotion  (031, 
036,  038). 

In  electrophysiological  experiments  where  a  tethered  fly  is  able  to  walk  on  a  trackball, 
by  monitoring  its  forward  speed  and  yaw  rotations  during  stimulated  with  visual 
motion  (Fig  13 A,  upper  panel)  we  found  that:  The  responses  of  the  LPTC  H2  (Fig 
13 A,  lower  panel)  to  temporal  frequencies  beyond  7  Hz  of  a  moving  patter  are 
elevated  when  the  fly  is  walking  (Fig  13B,  red  trace)  compared  to  those  in  stationary 
flies  (Fig  13B,  black  trace).  The  increased  responses  are  costly  for  the  fly  in  terms  of 
energy  consumption  but  indicate  a  necessary  adaptational  mechanism  to  adjust  its 
signalling  range  to  a  higher  dynamic  input  range  during  locomotion.  Food  deprivation 
abolishes  this  adaptation  in  a  gradual  way  (Fig  13C).  We  also  found  that  the  response 
increase  was  linearly  related  to  walking  speed  in  both  fed  (Fig  13D,  grey  line)  and 
food-deprived  flies  (Fig  13D,  black  line),  but  that  in  food-deprived  flies  the  increase 
was  systematically  reduced.  In  addition,  the  response  increase  was  found  to  be 
inversely  proportional  to  the  number  of  days  the  flies  were  starved  (Fig  13F).  Our 
findings  at  the  physiological  level  were  corroborated  by  studying  the  yaw  optomotor 
behaviour  in  flies,  using  wide-field  motion  for  visual  stimulation  (Fig  13E,  left  panel). 
In  fed  flies  the  temporal  frequency  tuning  of  the  bahaviour  for  frequencies  from  and 
above  4  Hz  was  high  and  stayed  at  a  similar  level  over  the  entire  dynamic  input  range 
(Figl3  E,  grey  trace  filled  circles).  Flies  starved  for  3  days  showed  a  significantly 
reduced  optomotor  response  (Fig  13E,  black  trace)  but  recovered  partly  after  1  day  of 
sucrose  feeding  (Fig  13E,  grey  trace  filled  triangles). 


Mean  spike  rate 


Walking  speed  (mms-^) 


6- 


Temporal  frequency  (Hz) 


0  12  3 

Time  starved  /  days 


Figure  13:  The  temporal  frequency  tuning  of  the  identified  H2-cell  depends  on  the  locomotor 
state  of  Calliphora  and  its  nutritional  state.  A:  Schematic  illustration  of  the  experimental 
setup.  B:  Temporal  frequency  tuning  in  walking  and  stationary  flies.  C:  Temporal  frequency 
tuning  in  fed  and  food-deprived  fies.  D:  Dependence  of  neural  response  on  walking  speed.  E: 
Schematic  illustration  of  optomotor  setup  (left)  and  optomotor  responses  in  fed  flies,  food- 
deprived  flies,  and  flies  which  were  fed  sucrose  after  3  days  of food  deprivation.  F:  Neuronal 
response  increment  due  to  adaptation  plotted  over  the  number  of  days  starved.  Note  that  the 
walking-dependent  adaptation  of  the  temporal  frequency  tuning  (panel  B)  is  reduced  due  to 
starving.  (Data  from  045). 
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Together  with  other  studies  on  the  olfactory  systems  in  starved  flies,  where  the 
activity  is  actually  increased  (rather  than  decreased),  our  results  suggest  that  under 
limited  energy  supply  adaptational  mechanisms  in  the  visual  system  are  reduced.  The 
interpretation  is  that  remaining  energy  resources  are  re-allocated  to  sensory  task 
which  increase  the  likelihood  of  survival  -  here:  olfaction,  which  aids  in  locating  food 
sources.  This  work  has  now  been  accepted  for  a  high  profile  publication  in  Current 
Biology  (045). 


Comparative  electrophysiology  on  Lobula  Plate  Tangential  Cells  in  dipteran  flies. 

As  mentioned  earlier,  to  identify  general  principles  and  species-specific  adaptations  of 
multisensory  integration  and  to  relationship  between  sensory  and  motor  coordinate 
systems  supporting  the  ‘Mode  Sensing  Hypothesis’  (042,  R4)  requires  to  study  of  a 
variety  of  different  dipteran  fly  species.  Ideally,  those  studies  would  include  a 
description  of  the  functional  anatomy  and  modelling  of  the  motor  systems, 
behavioural  experiments  and  an  assessment  of  the  neural  populations  involved  in 
optic  flow  processing.  The  four  families  of  dipteran  flies  we  planned  to  include  in  our 
study  are  shown  below.  They  all  have  different  morphological  and  anatomical 
features  likely  to  result  in  different  aerodynamic  properties,  a  pronounced  variety  of 
lobula  plate  tangential  cells  (LPTCs),  and  considerably  different  life  styles.  Together 
these  aspects  suggest  specific  relationships  between  sensory  and  motor  coordinate 
systems. 


Calliphoridae  Asilidae  Syrphidae  Tabanidae 


Due  to  the  limited  availability  of  the  different  species,  we  have  not  yet  been  able  to 
study  LPTCs  in  all  these  species.  Besides  the  detailed  knowledge  about  the 
subpopulations  of  LPTCs  in  Calliphora  -  the  reference  species  of  this  study  -  we 
were  only  able  to  record  form  a  number  of  optic  flow  processing  visual  intemeurons 
in  robberflies  so  far,  as  I  reported  earlier.  The  results  do  far  suggest  that  there  are 
indeed  species-specific  differences  in  the  response  properties  between  Calliphora  and 
the  xohhQrf[y  Philonicus  albiceps.  The  responses  of  LPTCs,  for  instance,  had  a  peak  at 
higher  temporal  frequencies  compared  to  Calliphora.  We  also  found  LPTCs  in 
robberflies  that  had  surprisingly  small  receptive  fields,  localized  in  the  fronto- 
equatorial  region  of  the  visual  field.  Cells  with  such  properties  in  Calliphora  had  not 
yet  been  reported  although  at  the  level  of  neck  motor  neurons  (026)  similar  visual 
receptive  field  sizes  were  found  which  potentially  support  some  sort  of  edge-detection 
mechanism  (e.g.  Hengstenberg  1991). 

Despite  those  some  differences  in  the  cellular  equipment  of  the  lobula  plate,  there  is 
one  type  of  LPTC  that  had  been  identified  in  Calliphora  which  was  found  in  all  other 
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elevation 


species  studied  so  far,  even  in  the  othopteran  locust.  A  receptive  field  organization 
similar  to  that  of  the  so-called  HI -cell  in  Calliphora  was  also  found  in  robberflies  and 
horseflies  (Figure  14). 
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Figure  14:  Fll-like  LPTCs  in  robberflies  and  horseflies.  The  panels  show  the  directional 
motion  preferences  and  sensitivities  of  motion  sensitive  interneurons  in  blowfly  (a),  robberfly 
(b),  and  horsefly  (c)  plotted  against  azimuth  and  elevation  within  the  right  visual  field.  LPTC 
of  the  type  -  which  has  been  identified  as  Hl-cell  in  blowfly  -  respond  to  back-to-front  wide- 
fiield  motion  in  the  right  visual  hemisphere.  In  Calliphora  the  Hl-cell  in  involved  in  detecting 
rotations  round  the  vertical  axis  (yaw). 


We  will  increase  our  efforts  to  investigate  LPTCs  in  hoverflies,  robberfies,  and 
horseflies  over  the  summer.  A  dedicated  rig  has  been  set  up  in  the  lab  that  allows  us  to 
perform  both  extracellular  and  intracellular  recordings  including  intracellular  staining 
to  identify  the  cell  under  study.  For  flexible  visual  stimulation  we  will  use  LED 
panels  as  described  in  one  of  my  earlier  reports.  In  addition  we  will  be  able  to  use 
another  rig  that  includes  a  back-projection  system  to  enable  wide-field  optic  flow 
stimulation. 


Fly-robot  interface  for  studying  multisensory  integration 

A  complementary  approach  to  study  multisensory  integration  has  been  developed 
over  the  last  three  years  that  utilizes  a  robotic  fly-brain  interface.  As  outlined  in 
earlier  reports  the  goal  is  to  use  neural  spiking  activity  recorded  on-board  from  an  Hl- 
cell  to  steer  the  robot  under  closed-loop  conditions.  To  do  so  we  had  to  miniaturize  an 
entire  electrophysiology  set  up  so  it  fits  onto  a  small  2-wheeled  robot  (Figure  15). 
This  process  has  now  been  completed.  Currently  we  are  working  on  a  control 
architecture  that  links  the  Hl-cell  signals  to  the  motors  driving  the  wheels.  A 
challenge  to  overcome  is  that  the  the  Hl-cell  increase  its  spike  rate  upon  back-to-front 
motion  (cf.  Figure  14).  Thus  during  forward  motion  of  the  robot  the  HI -cells  in  either 
side  of  the  visual  system  would  be  inhibited  and  could  not  be  used  to  control  the 
wheels  of  the  robot  in  order  to  avoid  collisions  with  obstacles  in  the  surroundings. 
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Figure  15:  A  robotic  fly-brain-interface  to 
study  multisensory  integration  in  Calliphora 
under  closed-loop  conditions.  A  small  two¬ 
wheeled  robot  (diameter  --  10  cm)  carries  a 
miniaturized  electrophysiology  setup.  Micro 
electrodes  will  be  positioned  within  the  lobula 
plate  using  mini  micromanipulators  mounted 
on  hollowed  bases  which  contain  shielded 
recording  electronics.  The  recoding  platform 
is  isolated  against  the  mechanical  vibrations 
caused  by  the  motors  by  a  layer  of  foam 
material.  The  fly  is  fixed  to  a  holder  mounted 
on  a  central  column.  Signals  from  the  HI -cell, 
a  spiking  LPTC,  are  extracellularly  recorded 
and  will  be  used  to  generate  motor  commands 
sent  to  the  motors  of  the  wheels.  A/D 
conversion  and  signal  processing  will  take 
place  on-board.  The  neural  signals  will  be 
used  in  a  control  architecture  to  avoids 
collision  with  obstacles  in  the  environment 
while  the  robot  is  freely  moving. 


This  problem  will  be  solved  by  implementing  a  eontrol  arehiteeture  that  exploits 
efferenee  eopies  (forward  models)  of  the  eommand  sent  to  the  motors  of  the  wheels 
foreing  an  undulating  trajeetory  of  the  robot.  The  efferenee  eopie  will  result  in  a 
predietion  of  the  neuronal  response  based  on  an  internal  model  of  the  relationship 
between  angular  veloeity  of  a  visual  pattern  and  the  Hl-eell  response.  The  predieted 
spike  rate  will  then  be  subtraeted  from  the  spike  rate  aetually  reeorded  from  the  Hl- 
eell  where  the  remaining  differenee  is  then  used  as  a  feedbaek  signal  to  eontrol  the 
motors  (Fig  16). 


Figure  16:  Control 
architecture  to  establish 
collision  avoidance  of  a 
robot  steered  by  a  fly. 

'Robot  model’  and  Hl- 
model  ’  refer  to  the 
relationship  between 
command  voltage  and 
angular  velocity  of  the 
robot  and  the 
relationship  between 
the  angular  velocity  of 
visual  motion  and  the 
spike  rate  of  the  Hl- 
cell.  (publication  in 
preparation). 


Sueh  strategy  requires  (a)  a  forward  model  of  Hl-aetivity  (Fig  17,  left)  and  (b)  the 
known  input-output  relationship  between  the  eontrol  voltage  sent  to  the  wheels  and 
the  angular  veloeity  of  the  robot  (Fig  17,  right).  For  the  arehiteeture  to  work  in  the 
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context  of  collision  avoidance  a  certain  response  properties  of  the  HI -cell  are  critical: 
close  objects  will  introduce  a  stronger  response  in  the  neuron  than  objects  at  larger 
distances  due  to  distance-dependent  decreases  in  contrast.  The  other  property  is  that 
the  response  of  the  HI -cell  is  a  monotonic  function  of  temporal  frequency  (Fig  17, 
left). 

Work  in  of  the  group  of  Prof  Egelhaaf  on  horizontal  LPTCs  suggested  that  so-called 
HS-cells  may  indeed  be  used  by  the  fly  to  assess  distance  -  although  only  during 
translational  phases  of  flight  following  fast  rotational  body  saccades.  A  similar 
strategy  of  ‘active  vision’  in  combination  with  an  efference  copy  as  described  above 
will  be  implemented  on  the  robot. 


L40FLY1 


Angular  velocity  (deg/s) 


Figure  17:  Angular  velocity  dependence  of  the  FI  1 -cell  response  in  the  lab  environment  (left). 
Note  the  monotonic  relationship  between  positive  angular  velocities  neuronal  spike  rate 
which  will  be  implemented  as  ‘Fll-cell  model’  in  the  closed-loop  control  architecture  in  Fig 
16.  Right:  Linear  relationship  between  the  command  voltage  and  the  angular  velocity  of  the 
robot  which  will  be  implemented  as  the  ‘robot  model  ’  in  the  closed-loop  control  architecture 
shown  in  Fig  16.  (publication  in  preparations). 


Eventually,  we  will  simultaneously  record  the  activity  of  other  LPTCs  of  which  we 
know  they  reflect  the  integration  from  sensory  modalities  other  than  motion  vision 
(e.g.  020,  034)  in  a  fly  that  is  actually  moving  in  space.  We  will  then  compare  the 
neuronal  responses  recorded  when  different  senor  systems  are  disabled  to  assess 
signal  integration  under  closed  loop  conditions. 
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2.4  Modelling  of  gaze  stabilization 


In  collaboration  with  Dr  Reiko  Tanaka  (Dept  Bioengineering)  we  have  been  working 
continuously  on  refinements  of  a  closed-loop  simulation  platform  that  models  the 
gaze  stabilization  system  of  Calliphora.  Most  modelling  approaches  were  based  on 
behavioural  experiments  (section  2.1)  and  assumed  -  in  a  first  approximation  -  a 
linear  time-invariant  system  (e.g.  C4).  One  major  result  of  this  approach  was  the  2 
degree  of  freedom  controller  discussed  above. 

We  have  recently  expanded  the  concept  of  the  2  degree  of  freedom  controller  to 
include  the  stabilization  of  the  body  against  unpredictable  external  disturbances.  It  is 
important  to  mention  that  the  diagram  shown  in  Figure  18  is  purely  conceptual  at  this 
point  in  time  and  has  not  yet  been  tested  experimentally  with  respect  to  the  flight 
control  component.  What  makes  it  plausible  in  terms  of  its  potential  implementation 
in  the  fly  nervous  system  is  that  the  same  pathways  known  to  control  the  animal’s 
gaze  are  also  thought  to  be  involved  in  flight  control.  The  core  functionality  of  the 
flight  control  system  is  therefore  likely  to  also  dependent  on  the  2  degrees  of  freedom 
controller  that  has  been  described  earlier. 


Figure  18:  Control  of  gaze  and  flight  by  a  2  degrees  of freedom  controller.  Upon  externally 
caused  body  rotations,  a  fast  feedforward  (FF)  signal  initiates  both,  compensatory  head 
movements  (HRJ  by  activation  of  the  neck  motor  (NM)  system  and  compensatory  body 
rotations  (BRJ  through  the  action  of  the  flight  motor  (FM).  This  fast  response  mediated  by 
the  haltere  system  reduces  the  retinal  slip  speed  and  thus  enables  the  processing  of  optic  flow 
along  the  slow  motion  vision  pathway  generating  a  feedback  signal  (FB)  that  contributes  to 
both,  HRc  and  BRc.  Any  remaining  misalignment  between  the  head  and  the  body  may  be 
measured  by  the  prosternal  and  chordotonal  organs  providing  slow  feedback  signals  (FB, 
grey)  primarily  to  the  neck  motor  system,  (unpublished  work  in  progress). 

The  control  diagram  in  Figure  18  only  considers  inner- loop  reflexes  induced  by 
external  perturbations.  During  voluntary  movements  of  body  to  change  the  trajectory, 
motor  commands  are  sent  to  both  the  neck  and  the  flight  motor,  and  efference  copies 
may  be  sent  to  the  sensors  supporting  the  feedforward  and  the  feedback  degrees  of 
freedom  so  that  the  sensory  signal  generated  as  a  result  of  voluntary  action  does  not 
immediately  cause  a  counteracting  stabilization  reflex.  Similar  schemes  in  spirit  have 
been  proposed  earlier  (R3)  and  Chan  et  al  1998.  But  firstly,  they  did  not  include  a  2 
degree  of  freedom  controller.  And  secondly,  they  are  only  weakly  supported  by 
experimental  evidence.  Further  closed-loop  simulations  including  the  scheme  will 
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have  to  show  (i)  its  viability  and  (ii)  may  inform  the  design  of  further  experimental 
studies  in  its  support  or  to  falsify  it. 

In  a  reeent  study  that  has  been  aeeepted  for  publieation  at  a  eontrol  engineering 
eonferenee  we  took  a  very  different  approaeh:  We  propose  a  novel  mathematieal 
framework  based  on  an  extended  Kalman  filter  and  a  eontroller  with  infinite-horizon 
dynamies  that  minimises  the  eosts  assoeiated  with  musele  eontraetion  on  the  one  hand 
side  and  imperfeet  gaze  stabilization  on  the  other  (C9).  In  elosed-loop  simulations  the 
model  explieitly  eonsiders  inherent  eonstraints  in  biologieal  systems,  i.e.  ambiguity 
and  noisiness  of  sensor  signals,  inevitable  response  delays  along  sensory  pathways, 
and  limited  energy  supply  for  both  the  neek  motor  and  the  nervous  system.  Figure  19 
shows  the  bloek  diagram  of  the  system. 


Figure  19:  Gaze  stabilization 
system  based  on  a  Kalman 
filter-type  state  estimator  and 
a  controller  providing 
optimized  sensoy  feedback  to 
the  motor  plant.  The  scheme 
involved  the  motion  vision 
pathway  provided  by  the 
compound  eyes,  halteres, 
ocelli  and  neck  proprioceptors 
which  measure  external 
disturbances,  dXf^\  Small 
deviations  of  from  a 
horizontal  head  orientation 
are  tolerated  due  to  the  cost 
associated  with  muscle 
contraction,  (from  C9) 


Although  it  is  unlikely  that  flies  have  evolved  an  implementation  of  an  optimized 
Kalman  filter  in  their  nervous  system  the  proposed  framework  may  as  well  be  of 
interests  for  teehnieal  applieations  ineluding  autonomous  miero  air  vehieles. 
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3.  Conclusion: 


Within  the  period  of  funding  we  have  made  significant  progress  in  all  areas  of 
research  carried  out  in  my  lab  and  on  the  basis  of  collaborations.  The  collaboration 
with  Sean  Humbert  turned  out  to  be  very  successful  with  us  meeting  at  least  once  or 
twice  a  year,  being  in  regular  email  contract  and  talking  on  the  phone  whenever  some 
immediate  discussion  on  our  collaborative  work  was  required.  We  published  together 
on  3  occasions  and  are  still  working  together  on  a  first  paper  where  the  modes  of 
motion  in  Calliphora  are  related  to  the  preferred  rotation  axes  of  its  VS -cell  in 
addition  to  another  conference  publication  on  a  bioinspired  ocellar  sensor. 

We  also  engaged  on  another  grant  proposal  to  AFOSR/EOARD  together  with  Graham 
Taylor  on  the  ‘Mode  Sensing  Hypothesis’  which  is  a  key  element  of  the 
implementation  road  map  of  the  AFRL-DSTL  programme  arrangement  on  bio¬ 
inspired  technologies.  The  start  of  the  grant  had  been  significantly  delayed  due  to  the 
financial  situation  in  the  US.  Meanwhile  funding  for  the  first  year  has  been  set  up  at 
the  University  of  Maryland. 

One  aspect  regarding  the  scientific  programme  of  this  grant  that  had  been 
underestimated  was  the  impact  of  limited  availability  of  dipteran  species  which 
cannot  be  bred  in  the  lab.  Although  we  developed  strategies  to  acquire  other  species 
by  means  of  wild  catches  and  temporary  breeding  of  hoverflies  except  for  Calliphora 
we  were  quite  often  short  of  experimental  animals.  This  was,  however,  compensated 
for  by  many  complementary  approaches  and  alternative  aspects  sensorimotor  research 
so  that  a  constant  stream  of  high  profile  output  was  produced  in  my  lab  during  the 
funding  period. 

Another  limiting  factor  certainly  is  a  comparatively  small  size  of  my  research  group.  I 
mentioned  in  my  previous  reports  already  that  the  comprehensive  approach  1  apply  to 
insect  sensorimotor  control  at  four  different  system  levels  and  across  several  species 
would  definitely  benefit  from  a  group  size  twice  as  large  as  my  own.  This,  however, 
would  require  me  to  write  way  more  grant  applications  with  I  would  like  to  do,  but 
cannot  because  of  a  limited  time  budget. 

In  an  ideal  scientific  world  and  to  facilitate  research  along  the  merits  of  AFRL  and 
AFOSR  stronger  support  in  terms  of  finances  would  be  much  appreciated.  Having 
said  this,  it  is  very  clear  that  I  would  not  have  been  able  to  engage  on  a  more 
comprehensive  approach  on  sensimotor  control  without  the  continuous  funding 
stream  provided  by  AFRL/ AFOSR  -  normally  through  BOARD. 

Irrespective  the  limitations  mentioned  so  far,  we  have  made  excellent  progress  in 
particular  with  work  on  the  3-  and  4-dimensional  functional  characterization  of  the 
neck  and  flight  motor  systems,  work  that  was  initiated  originally  by  Daniel  Schwyn 
and  Martina  Wicklein  in  my  group.  The  work  on  state-dependent  processing  which  is 
highly  relevant  to  the  topic  of  energy  efficient  information  processing  was  also 
initiated  in  my  lab  by  Kit  Longden  and  has  developed  into  an  aspect  several  world¬ 
leading  labs  are  now  working  on.  And  finally,  the  quantitative  behavioural  work  and 
modelling  projects  in  my  lab  and  in  collaboration  with  others  scientists  has  reached  a 
continuous  level  of  comparatively  high  productivity. 


Holger  G  Krapp 


London,  16  March  2014 
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